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(57) Abstract 

An object oriented method of structuring an executable software step program creates programs by composing a set of software 
objects or steps (200) self-organized into a hierarchical structure which provides the execution order, the connection of inputs (510) to 
outputs (234) at each step and between steps, and output data transformations. Tlie step hierarchy also takes advantage of commonality 
between steps that perfomi similar operations. 'ITie method makes applications very easy to set up and run while ensuring that challenging 
applications can be provided and perfomied. Programs arc built from a recipe database or directly using an easy to use point and click 
graphical user interface. Ttic method also provides a mechanism for the step program to create the user interface components that allow 
the training and setting of parameters using the GUI (Graphical User Interface) (690) library native to the platform (computer or system) 
running the program. Thus, step programs, such as machine vision programs, can be run unchanged on different CPU architectures and 
lake advantage of image processing acceleration hardware when available. 
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OBJECT ORIENTED METHOD OF STRUCTURING A SOFTWARE STEP PROGRAM 


FIELD OF THE INVENTION 
This invention relates generally to computer programs and 
more particularly, to the object-oriented representation and 
architecture of such software programs and to the 
representation and properties of computer programs such as 
those programs used to describe and effectuate real time 
control of various manufacturing and process control 
applications . 

BACKGROUND OF THE INVENTION 

The ultimate goal of a computer (application) program 
is to control the operation or processing of some form of 
data. More specifically, the goal of a machine vision 
system application program is to extract important features 
from image data from which a description, interpretation, or 
understanding of the objects m'the image can be provided by 
the machine (computer) for the purpose of classification, 
reporting and decision making. 

In a machine vision application, for example, a set of 
operations or steps are executed in sequence. A typical 
machine vision program flow diagram is shown in Figure 1. 
The steps generally begin with the acquisition of an input 
image, which is captured by a camera and placed in memory, 
step 1. The image is optionally preprocessed, step 2, 
before the image is segmented and features are extracted 
from it. Segmentation and feature extraction, step 3, of 
the image into one or more distinct objects can be 
accomplished, for example by extracting their boundaries. 
Features can be any data that describes or identifies parts, 
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feat.ures (holes, etc.) and objects in the camera scene, such 
as size, area, length, distance or like parameters. 

Features are usually expressed in pixel units that 
describe the geometric position of a feature in the image 
5 relative to some fixed position in the image such as the top 
left corner of the image (also called an image coordinate 
system) , or in physical units' like ''mm" or inches (also 
called a camera coordinate system) if the position of the 
camera in the scene is known or can be measured (usually 
10 referred to as calibrating the camera) . These features 

are typically checked against tolerances or classified into 
various categories, step 4, before a decision, step 5, can 
be made or pass/fail status and results obtained, step 6. 
Reports can also be generated, detailing the processing 
15 application, step 7. External control inputs, step 8, are 
used to activate the tolerance checking and to provide 
nominal tolerance values during execution for a particular 
feature . 

SUMMARY OF THE INVENTION 
The object oriented method of structuring a 'software 
step program, which is the subject of the present invention 
and described herein, has many advantages such as it: 

1) Makes simple vision applications very easy to set up, 
train and run, while ensuring that complex applications 
can also be readily configured. 

2) Provides a way to extend the programs by providing drop-in 
functionality in the form of language independent software 
components. New machine vision operations can be added to 
the system arbitrarily as well as new user interfaces for 
them. There are no arbitrary limits on the type of 
operation that can be done or data that can be returned by 
these computer programs. 
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3) Supports building the programs from a recipe database or 
directly using an easy to use point and click graphical 
user interface. 

A) Allows access of any data ' in the step program by name for 
easy scripting either locally or over a network and free 
form expression support. 

5) Provides a mechanism " for' expressing result **data in the 
step program in different systems of coordinates so that, 
for example,, measurements between features ' in different 
cameras can be easily computed when the object is too big 
to fit in one camera image. 

6) Allows any such named data to be encoded in a form that is 
compatible with the Internet addressing scheme known as 
the Uniform Resource Locator (URL) address for access to 
any data either locally or from remote host machines on an 
Internet or Intranet network. 

7) Provides a mechanism for the machine vision program to 
create the user interface components that allow the 
training and setting of parameters using a GUI (Graphical 
User Interface) library native to the platform (computer 
or system) that the program is loaded on. 

8} Allows the training and tryout (i.e. trial runs) of the 
machine vision program on a host with the vision processor 
plugged into a peripheral extension bus of the same host 
computer or plugged into' the peripheral extension bus of a 
different host computer that is on the same network as the 
setup and training host. 

9) Allows the training and tryout of ^ the machine vision 
program on a host computer with a standalone vision 
processor connected to the host system network. 

10) Runs the machine' vision computer programs unchanged on 
different CPU architectures and takes advantage of image 
processing acceleration' hardware when available. More 
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specifically the architecture supports a variety of 
processing models (corresponding to the runtime 
configurations as disclosed in commonly-owned, co-pending 
U.S. Provisional Patent application No. 60/066,339, filed 
5 November .21, 1997, which is incorporated herein by 

reference), including: 

a). Execurion of the program on a vision processor board 
. . thq. . target ) plugged in to a peripheral extension bus 
of the host computer. 

10 b) Execution of the program on the host using a vision 

board plugged into the extension bus for image 
acquisition and accelerated image processing. 

c) Execution of the program on the host using a vision 
board for image acquisition (frame grabber). 

15 d) For each output or result generated, a system of 

coordinates which specifies the units this output is 
expressed in is attached. 

e) Execution, of -the program on a standalone vision 
processor system (the target) connected to the host 
20 , over a network. 

10) Allows machine vision programs- to also be monitored 
once running on the vision processor board or vision 
processor,, ' .target for the purpose of diagnostics and 
debugging from a local- host or a remote host. 

25 The' invention clescribes an object oriented method of 

structuring a software step * program, which creates machine 
vision' programs by 'composing a set of 'software objects or 
steps self-organized into a hierarchical structure which 
both provides the execution, order and . the connection of 

30- inputs\.to outputs at- each step and between- steps . 

Steps' are- alsoi -organized into an -object-oriented 
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software hierarchy to take advantage of commonality between 
steps that perform similar operations. 

A" step program encodes a wealth of information. For 
example, a step program: 
5 a) contains a list of operations that together make up the 
machine vision application; 
b) for each step (operation), encode's the* set of parameters 
and settings required" to execute ' tha^t"" step (operation) 
successfully; 

10 c) for each step (operation)',' defines the -inputs that the 
step (operation) accepts and the' outputs or results that 
are generated; 

d) for outputs that are generated, attaches a system of 
coordinate information so that results can be expressed in 
15 real physical units; 

e_)_ contains information that allows for the calibration of 
a system of coordinates to physical, real-world units 
like inches or mm . and also allows the conversion of 
.-results from one system of coordinates to another; 

20 fj_ defines the physical relationships between a system of 

coordinates by organizing them into a hierarchical tree 
structure referred to as a" part step tree, which 
allows, for example the description- of . a moving camera 
by inserting -a stage pa'rt -step :( i . e . ' the motion device 

25 the camera is mounted on) between the camera system of 

coordinates and .. the .world system of coordinates 
(usually the table or.,^ fixture the entire, apparatus is 
bolted on) ;. . . . . , 

g) contains information: : to detect whether a particular 

30 operation needs to be setup/ or trained .by the operator of 

the system before it.ican run successfully; 
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h} for -each operation, contain-s information to invoke the 
required user interface components that assist an operator 
wher. setting up and training step programs regardless of 
the GUI (Graphical User Interface) environment the step 
5' - , program is ; loaded on; 

i) for each operation, can construct an executable 
' coxputer- program that runs on the host PC or on a vision 
:.3rqct board directly and which 'dynamically detects 
hardware and uses hardware acceleration when available; 
10 and 

-] ! naxer for each step operation as well as inputs, 
GU-C',;tL , i;ettings and parameters, which provides a 
stra- qr. t ijrv;ard mechanism for accessing any results in the 
step .c-roqrax by name from an expression, spreadsheet or 
15 script. 

In addition to describing . sequences of vision 
operations,, the disclosed step program architecture can also 
be used to describe, a variety of other cell control 
operations which often have to be performed in conjunction 
20 with vision processing. -Such programs include but are not 
limited ■ to • I/O, communications, and . motion control 
applications. The step program architecture can even be 
used to- exclusively describe such cell control programs that 
do not even contain any vision processing . at all. 


-25 BRIEF-. DESCRIPTION OF THE DRAWINGS 

- These and - pther -f eatures ■ and advantages of the present 
•invention will be better understood by reading the following 
^ detailed description, _ taken together with the drawings 
- wherein : , . . . . 

30* Figure 1 shows - the typical software components that 

make -up a machine . vision, program; - 
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Figure 2 shows a sample user interface view of a step 
program within a Host Graphical User Interface window; 

Figure 3 shows ' how the step program of fig 2 is 
executed and in what order steps Yun; 

Figure 4 shows the internal- " step object representation 
that supports this execution model; . 

Figure 5 details the flow of data within a step when it 

is executed; .. .... - 

Figure 5 shows the , ob j ect-oxiented inheritance 
hierarchy that supports both the step execution and data 
flow models; 

Figure 7 shows the object-oriented inheritance 
hierarchy that supports the step execution late binding for 
optimal performance; 

Figures 8A-8C are block schematic diagrams showing the 
host and target system network connectivity that are 
supported by the object oriented method of the present 
invention, for' the - purpose of creating,- setting up • and 
trying out an applications program such as -a machine vision 
program; and 

Figure 9 shows the relationship between the output 
results generated by a- step program arid system of 
coordinates -in the context of a two camera, sample step 
program which calculates the' distance 'between features in 
separate camera images, 

.. DETAILED . DESCRIPTIQN QF .THE-:- PREFERRED EMBODIMENT 
-The machine vision step program ' architecture that is 
the subject of this invention- is- preferably implemented by a 
set of software objects called - -'steps^ organized in a 
hierarchical tree structure. This representation lets 
client -"code treat obj-ects* • and composition of objects 
uniformly. A sample - tree- structure representation of a step 
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program is shown in Figure 2. 

Step Program Object Framework 

'As described herein for exemplary purposes, steps are 
"software-- objects (iie/'C++ Classes) -which are combined to 
5 form the step program. They are organized in a tree or 
hierarchical ■■fashion * where ■ steps can' contain other steps 
•"{cafled children - step'S) to perform training, preprocessing, 
coordinate transformation and post processing operations. 
Steps communicate- parameters and results by using datum 
10 objects . - ' 

Datum' objects are software objects- that provide 
standard types for ail' results found in • a vision application 
and allow results from one 'step (output) to be used as 
inputs to other steps'. Steps and datum objects (datums) can 
15 be edited graphically by using standard GUI component object 
editors well Icnown -in the industry such as one or more of 
the following generic component type editor such as tree 
control, 'list control, text- control and/or 'picture control. 

Editors are software -objects that can create all the 
20 necessary views and user interface controls :'to change steps 
and datums using a point-and-click user interface. Figure 2 
shows ^ the editor object that is created to display an entire 
step program graphically in a tree control window. 

In addition, each step and datum in "step program 
25 contains a special identifier or " GUID- (Globally Unique 
IDentifier) which allows the- step or datum - to dynamically 
"create' the editor' objects that let a user change parameters 
and settings artd train a step or datum. " The GUID is stored 
on- the host in a special database sometimes - referred to as a 
30 ''registry" in the Microsoft Windows operating system, as 
well as insYde the step and datum object itself. 

The host - provides standard operating system APIs 
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(application programmer interfaces) to create an object 
dynamically given its GUID, This, in essence generates a 
dual' obfect hierarchy .where. . every runtime step or datum may 
contain a pointer to an, editor obj ect , providing a user 

■5 interface for manipulating the step parameter:S .and inputs. 
These editor objects on the host in turn pr.eate software 
components which in the preferred embodiment are implemented 
as ActiveX controls for Windows95.. or. , NT .operating systems, 
that provide familiar ■ views, windows and/or . dialogs for 

10 editing and training the step par.amete.rs . . 

On the target, these identifiers are either ignored 
when the step, program is controlled by the host or are used 
to generate runtime user interface editors to manipulate the 
step inputs when the machine vision step program is running 

X5 in a standalone configuration (as described in next 
section) . 

Some steps create other steps to handle the runtime 
processing and setup or training. These steps are 

particularly useful, in ..that they can assist the operator at 
20 setup time with intelligent training. Intelligent training 
• is the ■ operation .wherein the step is shown a single sample 
image representative .of the. features that need to be 
detected during runtime and perform a training operation to 
self-define the optimal set of runtime settings . and 
25 • parameters . 

The execution order is encoded by the step program 100 
first by the position of the step in the tree, i.e. in the 
order one would enumerate them by reading .their name from 
left-to-right- and top-to-bottom. . This is referred as 
30 walking or traversing, the tree making sure each position is 
■visited once- until all s.tep.s have been traversed. 

The execution- order... is , further encoded by a special 
attribute attached to each... step in the* tree, which can 
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modify the rule presented above. For example, step 112 in 
step program 100 . may have an attribute set which causes it 
to be executed before step 110, therefore changing the 
execution order as defined by the tree traversal rule 
mentioned above.., .This is called a preprocessing step or 
more , rigorously a step that has the preprocessing attribute 
turned . o,n.. As will be described .later, attributes are not 
r.es^tricted to being execution order modifiers. 

In Figure. .2, , each iconic .picture represents an 
individual step operation of step program 100 according to 
the present invention. Indented steps, in addition to their 
tree organization, have the following attributes: 

• Preprocessing steps, which are executed before their 
parent. {FindPinl, step 116, is an example of a 
preprocessing step) ; 

• Training steps, which assist the operator when setting up 
the parameters of an operation. (Find Setup, step 118, is 
an example of a 'training step); and 

• Post-processing steps, which are executed after their 
20 parent. (Warp Step, step 122, is an example of a post- 
processing step) . 

Therefore, the execution order is encoded by the step 
program 100 both by the position of the step in the tree and 
by the set and ' selected pre/post processing 'attribute as 
25 ' well. 

Special steps called container steps group together 
children steps and "set their execution order. These 
container steps may" also provide a common' input to all their 
children steps.' For* e":<ampie, ' the Snapshot step, step 110, 
30 defines the' input image that" ail its contained children 
steps, steps 112-'1'2'8 ,"" operate" on, while their order dictates 
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their rank of execution. 

Figure 3 shows the step program execution order of the 
step program 100 of Figure 2. The step program execution 
begins with the acquisition of an image, step 112. 
Following image acquisition, the program increrhents to the 
next step in the hierarchy/ which is the 2PinFind step, step 
114. However, since the 2PinFind step contains three 
contained children ' steps , FindPinl, FindPin2 ' arici- Warp Step, 
steps 116, 120 and 122', respectively, the program must 
inspect each child step's pre/post processing attribute in 
order to execute^ the steps in their proper order. 

In this ■ example/ since FindPinl and FindPin2, steps 116 
and 120, respectively, are tagged as preprocessing steps, 
they are executed before their parent step, which is the 
2PinFind step, step 114. (Note that when the FindPinl step, 
step 116, is executed, its contained child step, which is 
the FindSetup step, step 118, is entirely skipped since its 
SETUP attribute tag.s it as. a Train only step and not as a 
Runtime step.) 

The next step in the execution order is the parent 
step, step 114, the 2PinFind step. Following execution of 
the parent, step 114, the program executes all contained 
post-processing steps, which, in the present example 
includes the Warp Step, step 122, including its contained 
steps, which comprise the Defect Finder and OCV Fontless 
Tools steps, steps 124 and 126, respectively. Finally, the 
step program will execute . the I/O^ output step, step 128 and 
the step program will have been completed. 

Referring to the representation of the step program 
shown in Figure 2, it can be seen that, in addition to 
fixing the execution order, the children step, inputs are 
automatically connected to their parent container outputs. 

Three basic software objects encode this internal data 
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and execution flow. Figure 4 provides a detailed structure 
of these software objects. The objects comprise step 
objects, which may be parent objects or child objects, such 
as step objects 200 and 300. Additional classes of objects 
5 include reference objects, such as reference object 400 and 
datum objects, such as datum object 500. These objects are 
organized into an inheritance object-oriented hierarchy from 
.which other steps are derived from as shown in Figure 6. 

To further define the hierarchical structure, steps 

10 that are at the same indentation level, as shown in Figure 
2, (e.g. steps 112 and 114 or steps 124 and 126) are 
physically stored in a data structure called a map. This 
map storage is part of the parent step object, which 
contains all these children steps, i.e. the step from which 

15 these children steps are directly indented. A map is a 
standard C++ library container object that is used to store 
and link together a set of arbitrary objects. In addition, 
each object in the list can be individually tagged by an 
attribute. In the current invention, attributes are used to 

20 refine the execution order of steps (i.e. preprocessing and 
post processing) as well as mark special classes of steps 
(SETUP steps and PART steps) . 

Any step within a program contains a list of contained 
steps called a _children list (hereinafter childlist), 210 

25 of Figure 4. Every step in a ^childlist is tagged as 
.either: PRE (Pre Processing) (e.g. steps 211 and 212), SETUP 
(Train), (e.g. step 213), PART (Calibration, coordinate 
transformation), (e.g. step 215) or POST (Post Processing), 
(e.g. step 214), to control the order in which they should 

30 be executed with respect to their parent step. The 
_datumlist 230 contains a list of the step inputs (IN), 231 
and 232, outputs (OUT), 234 and resources (RES), 233. 
Resources (or resource datums) are parameters or settings 
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that need to be set in order for that step to perform its 
operation . 

In addition, parent objects include _reflists, 220, 
which list the set of objects that either are using this 
object datums or need to be notified when the object values 
are changed. This list typically contains editor objects, 
which provide the operator of the system with a graphical 
view of the step or datum on the screen. Figure 2 above is 
an example of such an editor object. 

Editors are created by using the _GUID (Globally Unique 
Identifier) , 240, stored inside every step or datum which 
defines the type of objecu that can provide user interface 
control for changing this step parameters and settings. 

During step execution, {Figure 5) all the datums tagged 
as IN datums 510, such as datums 1, 2 and 3 {512, 514 and 
516, respectively) are read by the step and, based on its 
internal resource datums, such as RES Datums 1, 2 and 3 
(235, 237 and 239) the output datums, such as OUT Datum 1, 2 
and 3 (236, 238 and 242) are calculated according to the 
operation that the step is performing. 

Each output datum has a pointer, which points to a 
special step object called a Part step. Part steps contain 
information about the units for output datums. A Part step 
is a member of a separate tree of objects, called a ''part 
tree", which is built when the step program is created. 
Special steps in the object framework, like the Snapshot 
step 110 'of Figure 2, create'^ when "they are inserted into 
the step program tree, additional steps to provide 
coordinate systems or units to their output datums. 

In ^ the case of a Snapshot step^ the output image is 
associated with this special Part step called a Snap Part 
step. The Snap Part step is stored' in the child list of the 
Snapshot step and given the attribute (PART) which flags it 
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as a system of coordinates type object. These special 
objects are not executed at runtime but support the 
Calibrate method once a camera has * physically been placed 
into position, either on a rigid stand for a fixed camera or 
5 ' on a motion stage for a moving camera. 

For the" step program shown on Figure 2, the part tree 
looJcs_like the following: 

+WorldPart << always there, World system of coordinate 
■ where everything" is measured from 
10 +SnapPart << each camera has a position in the World. 

+WarpPart << Some vision tool rotate and move the 
image mathematically, these generate system of coordinates 
as well. 

In the above image, the WorldPart represents the system 

15 of coordinates of the entire system with respect to some 
known physical position {e.g. a table, or a room in which 
the equipment is located) . This is a default system of 
coordinates which is created whenever a step program is 
loaded from disk or created using editor objects. 

20 The SnapPart represents the position of the camera in 

the world created by the Snapshot step 110 of Figure 2. 

The WarpPart represents how the image coordinates are 
transformed through the process of warping the image created 
by the WarpStep, step 122 of Figure 2. 

25 Each output datums, such as OUT Datum 1, 2 and 3 (236, 

238 and 242) includes "a part step (i.e. 501 of Figure 4) 
that contains a pointer to 'a system of coordinates (object) 
which encodes the units used by this output. For an output 
result which is, for example, 'the position of a feature in 

30 ah image (such as a hole) which is encoded by two numbers x 
and y, the system of coordinates consist of two axis where x 
represent the pixel"^ position along the horizontal axis of 
the image and y represents the pixel position of the feature 
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along the vertical axis of the image, with pixel position 
0,0 being the top-left corner of the image. 

That coordinate system object is, itself, a step that 
has the attribute "PART" active as shown at 501 in Figure 4. 
5 These Part steps are inserted in the_ step program whenever 
a step that generates an image as one of its result Output 
datum is created. 

For example, for the Step program 700 of Figure 9 which 
calculates the physical distance between features (in the 
10 present example "holes") in separate camera images, consists 
of two Snapshot Steps 702 and 704, each containing one 
preprocessing AcquireStep 706 to capture the image from the 
camera, and one postprocessing FindStep 708 to locate the 
position of each desired image attribute e.g. (hole) (x and 
15 y coordinates of the ''hole" center in each camera's image 
coordinate system) , each snapshot step 704 creates a 
SnapshotStep PartStep (i.e. Step with the attribute "PART" 
set) and inserts it into its children list _childlist as 
shown at 215 of Figure 4. The SnapPartStep encodes the 
20 position of the origin 0, 0 and, axis (horizontal and 
vertical) of the coordinate system attached to the Output 
image Datum of the particular SnapshotStep. It also encodes 
the unit of measurement that the features calculated from 
this image are reported in. 
25 Typically, all features and results extracted from a 

camera image are reported, in pixels until the camera is 
calibrated to inches or millimeters. The Output Image Datum 
of the SnapshotStep in turn contains a pointer to the 
SnapPartStep as shown in t.he Figure 4 datum list as output 
30 datum item 500d. 

Calibrating a camera is the process by which, once the 
camera position has been measured or is known a priori in 
the World (i.e. Camera 1 (710) and Camera 2 (712) Position 
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in the table coordinate system WorldPart 705 of Figure 9), 
position in a given camera image can be converted to 
position in the scene (the "world") . For the step program 
700 of Figure 9, the FindStep 708 reports the position (x,y) 
5 of each "hole" or other attribute in the image being sought 
after, in pixel coordinates as described earlier. 

The x,y position is then converted to world or scene 
, X,Y coordinates by each Snapshot 702/704 PartStep of the 
respective SnapshotStep output image Datum before the 

10 . physical distance between the two holes can be calculated. 
These twp coordinate systems are shown in Figure 9 as 
SnapPartCaml {XI, Yl) 710 and SnapPartCam2 (X2, Y2) 712. 

The Snap PartStep, when executed at runtime, converts a 
feature like the x,y position of each hole as in Figure 9 

15 from pixel coordinates to camera coordinates so that 
measuring the distance between the two holes, for example, 
can be calculated. In fact the SnapPartStep steps of the 
step program in Figure 9 convert the x,y locations of each 
hole to a system of coordinates common to all cameras (i.e. 

20 WorldPart cpordinate system 705 in Figure 9) . 

The WorldPartStep 705, Fig. 9, represents the system of 
coordinates of the entire system with respect to some known 
physical position like the physical table or object to which 
the two cameras are mounted on, and is created whenever a 

25 step program is loaded into the system either on a host 
computer or on a vision processor system. Once the position 
of each hole or other image attribute in the step program in 
Figure 9 has been converted by each SnapPartStep to a common 
system, of coordinates, (the WorldPart) the Pt-Pt Distance 

30. step 714 calculates the distance in WorldPart units 
(typically inches or millimeters) between the holes or other 
image attributes. 

The _refto field of every datum object (e.g. 622 of 
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Figure 6) handles connections between inputs of one step and 
outputs of another. This field always points to a datum of 
the same type that contains the values to be used by the 
step during the execution of the step program. 

Figure 6 describes the way the foundation objects are 
organized. All basic objects' can ' contain children objects, 
this is shown by the composite object, 600. Steps are 
conpcsitc objects that also contain datiims *" obj'ects 620, to 
receiv'- inputs and calculate output resulti : "Every datum 
object \ZZ contains one or more reference members 622 
( rcftv-: " that steps can ' be connected together through 
their -•.;-c;:tr. An output datum in this representation is a 
datu.T wr.ci^^ _retto pointer is pointing to itself. Editor 
object t.'O serves as input to Host display object 640. 

An .mmediate advantage of this structure is that a 
connected datum, i.e. a datum ' that is of the input or 
resource r.ype, can also keep its own value separate from the 
one that is used when running. This is normally used to 
store nominal values . 

Since every object in a step program is' named, it is 
possible to construct a string, which uniquely describes the 
position or address of a result datum in the program. This 
allows a scripting language, or a formula for a spreadsheet 
cell, to build complex expressions that manipulate step data 
directly after a step program has been executed. For 
example, in Figure 2, the FindPinl step, step 116, locates a 
pattern in the image and" returns its ' position as a point 
with two coordinates "(xVy}.' This output ' point is called 
^^bestmatch" inside the FindPinl step. To manipulate the 
point X coordinate from' a script,' an operator would simply 
type, ^^FindPinl.'BestPoint.x". {Note that 

"FindPinl . BestMatch" uniquely identifies this point in the 
Step program.) The full acidress ' that is guaranteed to be 
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unique is: 

"VisionBoardO . InspectionStep . SnapShotStep . 
2PinFindatep. FindStepl .BestPoint .X" . 

- .The expression. to .find out if this coordinate is 
greater or equal to . zero . is . 

J /'FindPinl .Be.stPoint ..X >= 0". 

Given this . representation and the fact that the host 
system is connected to the vision processor using either a 
local network (the vision processor on board the host system 
as shown in Fig. 8A) or a physical network as shown in 
Figures &B " and 8C, and also supported by co-pending U.S. 
Patent Application No. 09/026,052 and entitled Apparent 
Network Interface For And Between Embedded and Host 
Processors incorporated herein by reference, it is possible 
to encode the vision processor location into the datum 
address for the purpose of displaying on the host results 
that ' are calculated by a running machine vision step 
program. 

^ ThiS: representation relies on the fact that the vision 
processor board and/or host system is setup as a Web server 
and can . therefore answer connection . from a Web browser 
running on the local host system (Figure 8A) or a remote 
host system (Figures 8B and 8C) . 

The full address is concatenated with the Uniform 
Resource Locator address or URL of the vision processor 
board or vision processor system that is controlled by the 
host. This can be used to display within a Web page on the 
host the data value ' that is being computed by the machine 
vision step program executing on the vision processor. For 
*the express-ion *abbve, the "URL address is: 

http : //VisionProcessorNetworkName/exec/ShowValue? Inspections 
tep. . 7 , • . ^ . -f- - . 

SnapShotStep , 2PinFindStep . FindStepl . BestPoint . x 
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where : 

a) ' http : //VisibnProcessbrNetworkName/ identifies the 

vision processor board or system the host is connected to. 

b) exec/ShowValue specifies the program on the vision 

processor board or system to 'call " to get the value of the 
data calculated by the machine vision'step 

program. Inspect ionStep .Best Point .X is the address of the 
Datum in_the step program that needs to be accessed as 
described 'earlier . ■ . ■ ^ • ' i. . \- " 

■ Step Program Portable Execution 

Step programs can be hardware or software encoded and 
can be executed on different CPU architectures and can take 
advantage of hardware acceleration for image processing 
operations at runtime. • This is done by late-binding of a 
step program to a hardware platform upon which- it is to be 
run . .. - ' : ■ 

In an object-oriented language, : like C++, • every object 
that is allocated into memory also needs to be initialized. 
Calling a special ^function" ■ of - - the'-' object called a 
constructor initializes an ob j ect Because ■ the step code 
which implements the object functions or methods is part of 
the application code running on 'that CPU and nor part of the 
step program saved on disk, it is possible to implement the 
object or step functions differently depending on where the 
object or step is loaded. 

Figure 7 shows the ^techniques .^which are used at runtime 
to efficiently .execute a step program. ,. . , 

1. The root of the step, program, is, a special container 
object called a -Target object,-, which, ^ when constructed, 
i.e. when the vStepr prpgram . is- . loaded^ into memory 
causing the constructor method to be called, scans the 
computer for hardware'-'re^ourceS'. ' In" particular, image 
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acquisition devices as well as pipeline image 
acceleration hardware (as found on the Acuity Vision 
Processor board described in commonly owned, co-pending 
US Patent Application " Ser . No. 08/953,772, incorporated 
herein by' reference,' are detected. In addition, memory 
is allocated to receive the images digitized from the 
camera- in ,a- location where they can be efficiently 
analyzed - by-" the 'hardware . ' 

At runimo, the steps- 650, Figure 7, which perform image 
prccecsir.:: ■ operations construct low-level software objects 
.called r^acriLs -652a-652c, to - analyze the image. These 
sofuwarr- Ticcnts are created using a technique called a late 
binding nechod. Late binding is the process wherein the 
step Itself creates the software object (s) to perform the 
processing operation (i.e. the image processing step). Late 
binding is the process wherein executable code is 
selected/generated from a number "o"f possible executable code 
sets based upon certain characteristics such as physical CPU 
.present, operating system, type of memory, etc., with a view 
toward . maximizing performance. In the present invention, 
late binding is utilized to optimize performance of the 
image processing system based upon the physical location of 
the image in the image memory. 

As a result, the step program transparently supports a 
number of runtime models. For example: 

• The step program can be executed on a host CPU using 
digitizer hardware, such as the digitizer described in 
co-pending Provisional Patent Application No. 60/066,339 
and previously filed U.S. Patent Application identified 
as U. S. Patent Application No. 09/030,411 files Feb. 25, 
1998 and entitled FLEXIBLE PROCESSING HARDWARE 
ARCHITECTURE, both of which are incorporated herein by 
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reference, as a frame grabber board directly connected to 
the host CPU PCI bus. 

• The step p.rogram can be executed on a host CPU. However, 
image processing -.operations- .can be accelerated by using a 
vision processor such as described in commonly-owned, co- 
pending U.S. Patent Application No. 08/953,772 
incorporated herein by reference^. : - ^ r. 

• The' step program can be executed on- a target CPU board as 
described in commonly-owned, ^ co-pending U.S. Patent 
Application No, 09/030,411, with the board plugged inside 
the host CPU PCI bus; 

• The step program can be executed on a standalone target 
CPU board which is connected to the host CPU over a 
network or not connected at all, as described in the 
above-referenced U.S. Patent Application No. 09/030,411. 

Modifications and substitutions 'by one of ordinary skill 
in the art are' considered to be within the' scope of the 
present invention which ' is not to be limited except by the 
claims which follow. 

What is claimed is: 
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CLAIMS 

1. An object oriented method of structuring an 
executable step program comprising a plurality of software 
objects to facilitate the understanding, manipulation and 
executiqn of said step program, said method comprising the 
5 steps of : 

r • a) , defining a plurality of individual steps of said 
program, each of said plurality of steps including a set of 
. step program objects including inputs accepted by said steps, 
operations performed by said steps and outputs generated by 
10 said steps; 

b) naming every step program object included in said 
program; and 

c) organizing said step program objects into a 
hierarchical tree structure. 

15 2. The method as claimed in claim 1, wherein said step 

of organizing said step program objects further comprises 
grouping steps that perform related operations as related 
steps, including parent steps and contained children steps. 

3. The method as claimed in claim 2, wherein said step 
20 program objects comprise: steps, which are combined to form 

said step program and which comprise datums, including input 
datums, output datums and resource, datums; and editor 
objects, which crea^te. all views and user interface controls. 

4. The method as claimed in claim 3, wherein said 
25 editor objects create all views and user interface controls 

using a .Graphical .User Interface native to an operating 
. system running on a .computer upon which said executable step 
■ -program is loaded and, run.. 
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5. The method as claimed in claim 3, further 
comprising the step of: 

' assigning a globally unique identifier (GUID) to each 

step; and 

storing each GUID" in a database managed by' -an operating 
system on a system on which is to run the executable step 
program, as well as in each step and datum its^elf . 

6. The method as "claimed in * claim" 5 wherein said 
children steps are classified as: preprocessing ■ steps , which 
are executed before their associated parent step; training 
steps. which assist an operator in setting up step 
parameters; and post-processing steps, which are executed 
after their associated parent step. 

7. The method as claimed in claim 2, wherein said step 
of grouping said steps that perform related operations 
comprises including a list of contained steps in each parent 
step. 

8. The method as claimed in claim 1, further 
comprising providing at least one common input to more than 
one child step*. 

9. The method as claimed in claim 3'V wherein said 
resource datums comprise parameters that should be set in 
order to permit their associated steps to perform their 
operations. 

10. The method as claimed^ in" ctaim 1 further including 
the step of naming af 'Tea^st " one step program object 
generated by said step program' including combining a URL 
Internet address encoding to act as a unique address for 
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said at least one step program object which a step program 
generates . 

11. The method as claimed in claim 1, wherein said step 
of grouping said steps that perform related operations 

5 further comprises including a reference list in each parent 
step, which lists the objects that utilize a common datum or 
should be notified when object values are changed. 

12. The method as claimed in claim 11, wherein said 
grouping step further comprises including a reference member 

10 field in each 'datum object, which connects inputs of one step 
to outputs of another step. 

13. An object oriented method of structuring an 
executable vision system step program comprising a plurality 
of software objects to facilitate the understanding, 

15 manipulation and execution of said step program, said method 
comprising the steps" of: 

a) defining a plurality of individual steps of said 
vision system program, each of said plurality of steps 
including a set of step program objects including inputs 

20 accepted by said steps, operations performed by said steps 
and outputs generated by said steps; 

b) naming every step program object included in said 
vision system step program; and 

c) organizing said vision system step program objects 
25 into a hierarchical tree structure. 
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14. A method of structuring a software step program to 
provide enhanced portability of said step program, said 
method comprising the steps of: . 

including a special container step object in said step 
5 program, which scans a computer , upon which said program is 
loaded for available capabilities; 

automatically allocating image _memory in said computer 
in a preferred location to maximize runtime performance of 
said step program; and 
10 automatically creating the most efficient runtime code 

for executing said step program on said computer. 

15. The method as claimed in claim 13, wherein said 
step of automatically creating the most efficient runtime 
code comprises creating image processing steps using a late 

15 binding software programming technique to construct low-level 
software objects which process an image in the fastest manner 
possible based on the physical location of the image in 
memory. 
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